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Purpose/Agenda 


Purpose 

- Status update on the 
developing methodology to 
revise sub-system sparing 
targets. 

- To describe how to 
incorporate uncertainty into 
spare assessments and why it 
is important to do so 


- Demonstrate hardware risk 
postures through PACT 
evaluation 


Agenda: (list major topics of 
the presentation): 

- Background 

- Key Definitions 

- Overview 

- Problem Statement 

- Review of Current Process 

- Introduction of Proposed 
Process 

- Analysis Examples 

- Results 

- Forward Work 

- Backup -analysis results 



Background 



• Sparing Assessment 

- Currently, an annual assessment 
is performed to estimate the 
number of spares needed to keep 
the International Space Station 
(ISS) operational until 2020 (and 
beyond). 

- Historically, the Spares 
Assessment has not included all 
uncertainty, specifically epistemic 
(lack of knowledge). 

• Propose Methodologies to 
Supplement Sparing 
Assessment 

- Characterize and quantify 
uncertainty, at the Orbital 
Replacement Unit (ORU) level. 
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Key Definitions and Concepts 



• Bayesian Process 

- A probabilistic theorem to infer an adjustment to predicted (prior) statistic given historical data. 

- ISS specific application updates predicted an ORU failure rate given operational experience 
resulting in an operational (posterior) ORU failure rate. 

• Error Factor 

- A parameter of statistical distributions describing the variation and frequency of values. 

- An ISS application for Error Factor applies to estimated ORU failure rates. 

- The Error Factor weighs ORU operational experience against the predicted ORU failure rate. 

• Poisson Process 

- A statistical technique based on the Poisson probability distribution estimating the probability that 
ORU failures in a projected time period do not exceed the number of ORU spares. 

- Input parameters for the Poisson Process include: ORU annual failure rate, projected time period 
(vehicle life), and ORU current number of spares. 

• Lognormal Process 

- A statistical technique similar to the Poisson process but includes an additional parameter (error 
factor) determining the distribution variance in ORU failure rate. 
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Key Definitions and Concepts, cont.’d 



• Confidence 

- A measure of the fidelity of an estimate 

• Epistemic Uncertainty 

- “Epistemic uncertainty is due to a lack of knowledge about the processes, models, parameters, 
and behavior used in the analysis.” ( NASA Procedural Requirements 8705. 5A) 

- “The epistemic models deal with non-observable quantities. Failure rates and model 
assumptions are not observable quantities." (NASA/SP-201 1-3421) 

- Also called reducible uncertainty 

- Only epistemic uncertainty is reducible through operational experience 

• Aleatory Uncertainty 

- Intrinsic randomness of a phenomenon 

- Also called irreducible uncertainty 

- Can not be suppressed by more accurate measurements 

• Probability of Sufficiency (POS) 

- Likelihood that the number of current spares and/or proposed number of spares estimated to 
reach the end of life of the International Space Station (ISS) is less than or equal to the predicted 
number of ORU failures. 

- ORU POS can be calculated through the Poisson Process or Lognormal Process 

• Probability Target (PT) 

- Desired value in POS 

• Confidence Target (CT) 

- Desired confidence in POS value 
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Problem Statement 




There are two general shortcomings to 
the current annual sparing assessment: 

1 . The vehicle functions are currently 
assessed according to ‘confidence 
targets,’ which can be misleading. 

- Confidence calculations may be overly 
optimistic because they only take into 
account natural variability, i.e. 
randomness in times of failure. 

- Alternatively, for projections on 
necessary spares through vehicle life, 
current implicit function confidence 
targets may be overly conservative or 
optimistic. 

2. The current confidence levels are 
arbitrarily determined and do not account 
for epistemic uncertainty (lack of 
knowledge) in the ORU failure rate. 

- Due to inherent uncertainty, a more 
robust approach is warranted. 
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Overview 


Objective 

• Examine uncertainty, risk, and confidence for ISS Sparing 
Assessment 

- Explains how robust results can be obtained where there is lack of data. 

- There are two major categories of uncertainty that impact Sparing Assessment: 

• Aleatory Uncertainty: Natural variability in distribution of actual failures 
around an Mean Time Between Failure (MTBF) 

• Epistemic Uncertainty : Lack of knowledge about the true value of an 
Orbital Replacement Unit’s (ORU) MTBF 





Overview, cont.’d 




Approach 

• Going forward, the team plans to 
recommend changes to the spares 
confidence and assessment that 
will: 

- Take into account both types of 
uncertainty, 

- Show the dangers of not including 
epistemic uncertainty in sparing 
evaluations 

- Make recommendations that are 
realistic and show how robust 
results can be obtained 

- Expand the approach to the 
function level and to include 
minimization of spare costs 

• We therefore introduce a technique 
to include epistemic uncertainty 


Updated Approach: PACT 
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Current Process Flow 



ORU Probability 
of Sufficiency 
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Results of Current Process 
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• The Bayesian update process provides a posterior error factor (EF), which 
describes the epistemic uncertainty in the updated ORU failure rate and which is 
unused. 


• The Poisson process provides a probability that the number of spares exceeds 
the expected number of failures. 

- This estimate can be considered a point along a Y-axis 0-100% 

• When accounting for epistemic uncertainty in the ORU failure rate, we consider 
a corresponding “confidence value” in the probability of sufficiency estimate 

- This corresponding estimate is an added X-axis, 0-100% 

- Not including the epistemic uncertainty results in the sparing risk only being 
partially assessed 

• These two estimates make up a trade space, which we use to assess sparing 
risk and account for epistemic and aleatory uncertainty: 

- Probability of sufficiency (POS) 

- Confidence in the probability, 
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Current Model of ISS Sparing Risk 



ioo%| 


Accepted Risk 



o% 


50% 


100 % 


Un-quantified Confidence in POS 


Note: 50% is normally the minimum accepted confidence in the 
nuclear energy utility sector 
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Proposed Model for Sparing Risk: 

PACT Evaluation 





Methodology 


Model Parameters 


• Explicitly account for inherent epistemic 
uncertainty, lack of knowledge, in the 
ORU failure rate by utilizing the ORU 
posterior error factor from the Bayesian 
update process. 

• Use a Lognormal distribution to 
represent the uncertainty which allows 
use of the Bayesian posterior Error 
Factor 

- Higher values in the posterior error factor 
translate to greater levels of uncertainty in the 
ORU failure rate 


Input- ORU failure rate 
uncertainty distribution 

Output- values of the ORU 
Probability of Sufficiency (POS) 
with the associated, explicit 
Confidence level 


- Lower values in the posterior error factor 
translate to greater levels of certainty in ORU 
failure rate 
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PACT: Proposed Process Flow 





Bayesian 
Update Process 


Posterior ORU 
failure rate 
(1/MTBF) 


Log Normal 

Process 



ORU Probability 
of Sufficiency 


Confidence in 
the Probability 
Estimate 




PACT 


Page No. 13 








PACT Results 





• The regions describing current sparing risk- 1) accepted risk and 2) 
sufficiency, are no longer linear. The probability of sufficiency 
straight edge is now a curve. 

- However, the Lognormal with Error Factor = 1 equates to the Poisson 
process with no epistemic uncertainty 

- Error factor = 1 represents a scenario of absolute certainty in ORU failure 
rate. 


• Trade Space: Accounting for epistemic uncertainty, the trade-space is 
comprised of 4 regions of varying areas depending on the ORU 
sufficiency and level of epistemic uncertainty. 

• Area of Sufficiency 

• Area of Previously Accepted Risk 

• Area of Previously Accepted Risk that is now “sufficient” 

• Area of previously unidentified epistemic uncertainty 
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Probability of Sufficiency (POS) 



100 % 


0 % 


Introducing Uncertainty: PACT Model 

ofjSS Sparing Risk 




0 % 25 % 50 % 75 % 100 % 

Quantified Confidence in POS 


Note: 50% is normally the minimum accepted confidence in the 
nuclear energy utility sector 
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Benefits of the PACT 





• Using this model we can now say- 'we are n% confident that the 
ORU has m% probability of sufficiency through yearxxxx.’ 

- Importantly, the intersect of the curve and straight-edge represents the 
underlying confidence we have carried on ORU sufficiency 

• With this knowledge, we can assess the tradeoff along the curve 
between the desirable ORU Probability of Sufficiency (POS) value 
and our confidence in the value . 

- At Least Five Options: 

1 . We may accept the probability and our confidence as is. 

2. We may relax our confidence for a higher desirable Probability of Sufficiency 
(POS) value. 

3. We may consider a lower desirable Probability of Sufficiency (POS) value in 
favor of having higher confidence . 

4. We may begin discussion on procuring more spares to achieve a higher 
desirable POS value and associated confidence level. 

5. Change the projected horizon time for the analysis. 

- Lower confidence can be accepted in cases where there is a larger sparing supply and 
recovery capability. 

- Shorter time intervals would allow for closer tracking. 
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Testing PACT 
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• Objective: Selection represents a variety of ISS supportability challenges/risk 
impacting sparing levels 

- Demonstrates a breadth of uncertainty: empirical basis of ORU failure rate demonstrated 
versus lack of operational experience 


• Case Study: 6 ORUs were analyzed in detail to understand the utilization of the 
proposed process 

- RPCM Type 5 External -large population of installed units and spares providing rich 
operational experience including failures and redesign 

- IPEHG -small internal ORU, no operational experience 

- Pump Module -large external ORU, critical hardware with many hours of operation and a 
random failure 

- Hydrogen Dome -large internal ORU, no redundancy, relatively little operational experience 

- SARJ DLA / TRRJ DLA -external ORUs with redundancy, similar function differing failure 
rates, SARJ DLA modeled in the Usable Power function hierarchy, TRRJ DLA modeled as a 
separate ORU 


• Statistics: 

- The average error factor for entire population of ORUs: 3.89 (range: 1 .36 - 4.00) 

- ORU failure rate range: 1 .98E-12 to 3.5E-4 failures/yr 
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PACT Analyses Results 




ORU Name 

MTBF 

Installed 

Quantity 

kF 

Duty 

Cycle 

Total Expected 
Number of 
Failures through 
2020 

Current 
Number 
of Spares 

Posterior 

Error 

Factor 

POS 

through 

2020 

Confidence 
at POS 

RPCM T5 E 

268,443 

36 

1.2 

1 

12.69 

18 

1.36 

95% 

54% 

Improved Payload 
Ethernet Hub 
Gateway (IPEHG) 

50,719 

3 

1.31 

1 

6.11 

3 

4 

15% 

66% 

Pump Module 
Assembly (PMA) 

69,065 

2 

1.2 

1 

2.74 

4 

2.87 

86% 

63% 

Hydrogen Dome 

49,853 

1 

1.3 

1 

2.06 

4 

4 

95% 

66% 

SARJ Drive Lock 
Assembly (SARJ- 
DLA) 

278,241 

4 

1.2 

0.5 

0.68 

2 

4 

97% 

66% 

TRRJ Drive Lock 
Assembly (TRRJ- 
DLA) 

353,926 

4 

1.2 

0.5 

0.53 

1 

4 

90% 

66% 


Note: High confidence targets are based on the minimum system configuration for each ORU. 
The proposed approach will be useful in re-evaluating the POS and identifying a reasonable 
confidence target. 


Summary 





• A useful methodology has been proposed to supplement ISS 
sparing analysis. 

• Preliminary results indicate a valuable trade space for selecting 
optimal targets and identifying the confidence associated with the 
target. 
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Forward Work 



y^mr 


1. Extend the methodology to evaluate spare allocations for systems 
and functions to assure a desirable POS and confidence level for 
short term and long term trade-off 


2. Include the minimization of cost and resources while assuring a 
desirable POS and confidence level 


3. Evaluate robust techniques for determining desirable POS values 
and associated confidence levels and for handling different plausible 
error factors in component uncertainties 
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Backup 
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Probability of Sufficiency (PQS) (Failures ^Spares) 


Results: RPCM Type V Ext 




Confidence and Probability of Sufficiency through 2020 
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Results: IPEHG 
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Confidence and Probability of Sufficiency through 2020 
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Error Factor 1 (Poisson) 


Hydrogen Dome 




Confidence and Probability of Sufficiency through 2020 
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Probability of Sufficiency (POS) (Failures <5 pa res) 


SARJ DLA 




Confidence and Probability of Sufficiency through 2020 
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Pro lability of Sufficiency (POS) (Failures < Spa res) 


TRRJ DLA 




Confidence and Probability of Sufficiency through 2020 
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Error Factor 1 (Poisson) 


